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A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
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earned patent term adjustment. See 37 CFR 1 .704(b). 
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Disposition of Claims 

4) 13 Claim(s) 1-44 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) £3 Claim(s) l^fQ is/are rejected. 

7) Q Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
1 1 )□ The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§ 119 and 120 

1 3) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (f). 

a)D All b)D Some*c)D None of: 

1 .0 Certified copies of the priority documents have been received. 

2.0 Certified copies of the priority documents have been received in Application No. . 

3-D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. ^^^ks^f^ 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 12/jJ \ ^ : 
Attach ment(s) r n * 

1 ) K Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) Paper No(s). . 

2) [H Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) CD Notice of Informal Patent Application (PTO-152) 

3) D Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) D Other: 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 04-01) 



Office Action Summary 



Part of Paper No. 6 



Application/Control Number: 09/928,165 Page 2 

Art Unit: 2671 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351(a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21 (2) 
of such treaty in the English language. 

Claims 1 , 4-1 0, 1 7, 22, 23, 27, 28, 32, 35-36, and 40 are rejected under 35 
U.S.C. 102(e) as being anticipated by Heidmann et al. (6,057,833). 

As per claim 1, Heidmann teaches the claimed "method for enhancing video, 
comprising the steps of "identifying one or more positions in a first image" (Heidmann, 
column 12, lines 38-42); "converting said one or more positions to one or more locations 
in relation to a virtual surface, converting said one or more locations in relation to said 
virtual surface to one or more positions in a second image" (Heidmann, column 12, lines 
43-67); and "enhancing said second image based on said one or more positions in said 
second image" (Heidmann, column 14, lines 2-16). 

Claim 4 adds into claim 1 "using a pointing device to select one or more pixels on 
a monitor" which Heidman teaches in column 4, lines 47-48. 

Claim 5 adds into claim 1 "using a pointing device to select one or more pixels on 
a monitor and using said selected pixels to determine said one or more positions in a 
first image" which Heidmann teaches in column 4, lines 61-64, column 6, lines 23-28. 
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Claim 6 adds into claim 1 "selecting a region of said first image" which Heidmann 
teaches in column 6, lines 19-22. 

Claim 7 adds into claim 1 "said one or more locations are three dimensional 
locations on said virtual surface" which Heidmann teaches in column 12, lines 49-50. 

Claim 8 adds into claim 1 "said one or more locations are all on said virtual 
surface" which Heidmann teaches in column 12, lines 49-50. 

Claim 9 adds into claim 1 "said second image depicts an environment; said 
second image is captured by a camera; and said step of converting said one or more 
locations is performed without surveying a position of said camera in relation to said 
environment" which Heidmann teaches in figures 10A-10B. 

Claim 10 adds into claim 1 "said second image depicts an environment having an 
actual surface; said one or more positions in said second image are on an image of said 
actual surface in said second image; and said step of converting said one or more 
locations are performed without knowing distance between said first actual surface and 
a camera capturing said second image" which Heidmann teaches in col. 4, lines 4-8. 

Claim 17 adds into claim 1 "the steps of converting said one or more locations in 
relation to said virtual surface to one or more positions in a third image; and enhancing 
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said third image based on said one or more positions in said third image" which 
Heidmann teaches in column 9, lines 31-51. 

As per claim 22, Heidmann teaches the claimed "method for enhancing video" 
comprising the steps of "determining one or more locations in relation to a virtual 
surface" (Heidmann, column 12, lines 35-59); and "converting said one or more 
locations in relation to said virtual surface to one or more positions in a video image" 
(Heidmann, column 12, line 60 to column 13, line 25). 

Claim 23 adds into claim 22 "the steps enhancing said video image based on 
said one or more positions in said video image" which Heidmann teaches in column 8, 
lines 44-46. 

As per claim 27, Heidmann teaches the claimed "one or more processor 
readable storage devices" for storing processor readable code, said processor readable 
code for programming one or more processors to perform a method comprising the 
steps of "determining one or more locations in relation to a virtual surface" (Heidmann, 
column 12, lines 35-59); and "converting said one or more locations in relation to said 
virtual surface to one or more positions in a video image" (Heidmann, column 12, line 60 
to column 13, line 25). 
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Claim 28 adds into claim 27 "the steps of: enhancing said video image based on 
said one or more positions in said video image" which Heidmann teaches in column 8, 
lines 44-46. 

Claim 32 adds into claim 27 "the steps of storing one or more positions in a first 
image, said step of determining includes converting said one or more positions to said 
one or more locations in relation to said virtual surface; and enhancing said video image 
based on said one or more positions in said video image" which Heidmann teaches in 
column 4, lines 61-64, column 6, lines 23-28. 

As per claim 35, Heidmann teaches the claimed "apparatus for enhancing video" 
comprising "video modification hardware" (Heidmann, column 12, lines 35-59); and one 
or more processing devices in communication with said video modification hardware, 
said one or more processing devices perform a method comprising the steps of 
determining one or more locations in relation to a virtual surface, and converting said 
one or more locations in relation to said virtual surface to one or more positions in a 
video image (Heidmann, column 12, line 60 to column 13, line 25). 

Claim 36 adds into claim 35 "the steps of enhancing said video image based on 
said one or more positions in said video image" which Heidmann teaches in column 8, 
lines 44-46. 
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Claim 40 adds into claim. 35 "the steps of storing one or more positions in a first 
image, said step of determining includes converting said one or more positions to said 
one or more locations in relation to said virtual surface; and enhancing said video image 
based on said one or more positions in said video image" which Heidmann teaches in 
column 4, lines 61-64, column 6, lines 23-28. 



The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f)or (g) 
prior art under 35 U.S.C. 103(a). 



Claims 2, 3, 11-16, 18-21, 24-26, 29-31, 33-34, 37-39 and 41-44 are rejected 
under 35 U.S.C. 103(a) as being unpatentable over HEIDMANN et al. (6,057,833). 
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Claim 2 adds into claim 1"said virtual surface is a sphere" which would have 
been obvious in view of Heidmann's projection surface (col. 12, lines 49-50) because 
such surface can be any type of surface such as a sphere. 

Claim 3 adds into claim 2 "said first image and said second image are video 
images; said first image depicts an environment having an actual surface (Heidmann, 
e.g., play field); said second image depicts said actual surface (Heidmann, e.g., camera 
field); said one or more positions in said first image are on an image of said actual 
surface in said first image; said one or more locations are three dimensional locations 
on said sphere; said one or more positions in said second image are on an image of 
said actual surface in said second image (Heidmann, figures 10A-10B); said step of 
enhancing includes editing said second image in relation to said one or more positions 
in said second image, said step of converting said one or more locations is based on 
camera sensor data; and said camera sensor data pertains to an attitude of a camera 
capturing said second image" which would have been obvious to Heidmann's camera 
shot on a play field (column 12, line 35 to column 13, line 33) because the 3D surface 
can be any type of 3D surface such as a sphere. 

Claim 1 1 adds into claim 1 "said step of converting said one or more locations is 
based on camera sensor data; and said camera sensor data pertains to an attitude of a 
camera capturing said second image" which would have been obvious to Heidmann's 
camera data (column 4, lines 4-8; column 12, lines 43-57) because the camera data can 
be collected through the sensors as claimed. 
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Claim 12 adds into claim 1 "said step of converting said one or more locations is 
based on camera sensor data and stabilization offset data; said camera sensor data 
pertains to an attitude of a camera capturing said second image; and said stabilization 
offset data is used to correct said camera sensor data" which would have been obvious 
to Heidmann's camera on the play field because the accuracy of data is necessary for a 
comprehensive display of the scene (Heidmann, column 4, lines 4-15 or figs 10A-10B). 

Claim 13 adds into claim 12 "the steps of identifying positions of edges; 
transforming said positions of said edges to edge locations on said virtual surface; 
transforming said edge locations to edge positions in said second image using said 
camera sensor data; determining said stabilization -offset data based on a difference 
between said transformed edge positions in said second image and actual edge 
positions in said second image" which would have been obvious to Heidmann's camera 
on the play field because the accuracy of data is necessary for a comprehensive display 
of the scene (Heidmann, column 4, lines 4-15, column 6, line 53 to column 7, line 27). 

Claim 14 adds into claim 1 "said step of enhancing includes adding an 
enhancement image to said second image and accounting for occlusions of said 
enhancement image" which would have been obvious in view of Heidmann's 
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transparency scene (col. 6, line 64 to column 7, line 1 ) because the opacity and/or 
transparency of the display scene implies the occlusions of the inserted object. 

Claim 15 adds into claim 14 "said step of accounting for occlusions of said 
enhancement image includes accessing a color map and editing pixels of said second 
image according to said color map" which Heidmann teaches in col. 5, lines 63-65 in 
which the attribute data of the inserted object can be any well known type of 3D object 
attribute such as color, texture, depth, ... 

Claim 16 adds into claim 14 that "said second image is an image of an 
environment having an actual surface; and said enhancement image appears to be on 
said actual surface" which Heidmann teaches in figures 10A-10B. 

Claim 18 adds into claim 1 "said first image and said second image are video 
images; said first image depicts an environment having an actual surface; said second 
image depicts said actual surface; said step of identifying includes annotating an image 
of said actual surface in said first image, said one or more positions represent said 
annotation, said annotation is at a first orientation in relation to said actual surface; and 
said step of enhancing includes adding said annotation to an image of said actual 
surface in said second image at said first orientation in relation to said actual surface" 
which would have been obvious to Heidmann's inserted objects and animation (column 
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3 line 56 to column 4, line 14) because one of those inserted objects can be an 
annotation. 

Claim 19 adds into claim 18 "said step of annotating an image includes receiving 
a set of points from a touch screen" which Heidmann teaches in column 8, lines 44-46. 

Claim 20 adds into claim 1 "said step of identifying one or more positions a first 
image include identifying a position to add a virtual advertisement; said step of 
converting said one or more locations includes determining where to add said virtual 
advertisement into said second image; and said step of enhancing includes adding said 
virtual advertisement into said second image" which would have been obvious to 
Heidmann's inserted objects (column 4, lines 1-14) because the inserted object can be 
"a virtual advertisement". 

Claim 21 adds into claim 1 "said step of identifying one or more positions in a first 
image include identifying a set of positions in an environment, said set of positions 
bound a virtual advertisement; said step of converting said one or more positions 
includes transforming said set of positions in said environment to locations on said 
virtual surface; said step of converting said one or more locations includes determining 
where said set of positions are in said second image; and said step of enhancing 
includes adding said virtual advertisement into said second image where said set of 
positions are in said second image" which would have been obvious to Heidmann's 
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inserted objects (column 4, lines 1-14) because the inserted object can be "a virtual 
advertisement" which can be added to the transformed scene. 



Claim 24 adds into claim 23 "said step of enhancing includes adding an 
enhancement image to said video image and accounting for occlusions of said 
enhancement image" which would have been obvious in view of Heidmann's 
transparency scene (col. 6, line 64 to column 7, line 1) because the opacity and/or 
transparency of the display scene implies the occlusions of the inserted object. 

Claim 25 adds into claim 22 "said virtual surface is a sphere; and said one or 
more locations are three dimensional locations on said sphere" which would have been 
obvious to Heidmann's camera shot on a play field (column 12, line 35 to column 13, 
line 33) because the 3D surface can be any type of 3D surface such as a sphere. 

Claim 26 adds into claim 22 "said video image depicts an environment; said 
video image is captured by a first camera; and said step of converting said one or more 
locations is performed without knowing a position of said first camera in relation to said 
environment" which would have been obvious to Heidmann's camera on the play field 
because the accuracy of data is necessary for a comprehensive display of the scene 
(Heidmann, column 4, lines 4-15, column 6, line 53 to column 7 f line 27). 
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Claim 29 adds into claim 28 "said step of enhancing includes adding an 
enhancement image to said video image and accounting for occlusions of said 
enhancement image" which would have been obvious in view of Heidmann's 
transparency scene (col. 6, line 64 to column 7, line 1 ) because the opacity and/or 
transparency of the display scene implies the occlusions of the inserted object. 

Claim 30 adds into claim 27 "said virtual surface is a sphere; and said one or 
more locations are three dimensional locations on said sphere" which would have been 
obvious to Heidmann's camera shot on a play field (column 12, line 35 to column 13, 
line 33) because the 3D surface can be any type of 3D surface such as a sphere. 

Claim 31 adds into claim 27 "said video image depicts an environment; said 
video image is captured by a first camera; and said step of converting said one or more 
locations is performed without knowing a position of said first camera in relation to said 
environment" which would have been obvious to Heidmann's camera on the play field 
because the accuracy of data is necessary for a comprehensive display of the scene 
(Heidmann, column 4, lines 4-15, column 6, line 53 to column 7, line 27). 

Claim 33 adds into claim 32 "said first image is a video image; said first image 
depicts an environment having an actual surface; said video image depicts said actual 
surface; said step of storing includes storing an annotation of an image of said actual 
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surface, said one or more positions represent said annotation, said annotation is at a 
first orientation in relation to said actual surface; and said step of enhancing includes 
adding said annotation to an image of said actual surface in said video image at said 
first orientation in relation to said actual surface" which would have been obvious to 
Heidmann's inserted objects and animation (column 3 line 56 to column 4, line 14) 
because one of those inserted objects can be an annotation. 

Claim 34 adds into claim 32 "said step of storing includes storing a set of 
positions in an environment, said set of position bound a virtual advertisement; said step 
of converting said one or more positions includes transforming said set of positions in 
said environment to locations on said virtual surface; said step of converting said one or 
more locations includes determining where said set of positions are in said second 
image; and said step of enhancing includes adding said virtual advertisement into said 
second image where said set of positions are in said second image" which would have 
been obvious to Heidmann's inserted objects (column 4, lines 1-14) because the 
inserted object can be "a virtual advertisement". 

Claim 37 adds into claim 35 "said step of enhancing includes adding an 
enhancement image to said video image and accounting for occlusions of said 
enhancement image" which would have been obvious in view of Heidmann's 
transparency scene (col. 6, line 64 to column 7, line 1) because the opacity and/or 
transparency of the display scene implies the occlusions of the inserted object. 
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Claim 38 adds into claim 35 "said virtual surface is a sphere; and said one or 
more locations are three dimensional locations on said sphere" which would have been 
obvious to Heidmann's camera shot on a play field (column 12, line 35 to column 13, 
line 33) because the 3D surface can be any type of 3D surface such as a sphere. 

Claim 39 adds into claim 35 "said video image depicts an environment; said 
video image is captured by a first camera; and said step of converting said one or more 
locations is performed without knowing a position of said first camera in relation to said 
environment" which would have been obvious to Heidmann's camera on the play field 
because the accuracy of data is necessary for a comprehensive display of the scene 
(Heidmann, column 4, lines 4-15, column 6, line 53 to column 7, line 27). 

Claim 41 adds into claim 40 "said first image is a video image; said first image 
depicts an environment having an actual surface; said video image depicts said actual 
surface; said step of storing includes storing an annotation on an image of said actual 
surface in said first image, said one or more positions represent said annotation, said 
annotation is at a first orientation in relation to said actual surface; and said step of 
enhancing includes adding said annotation to an image of said actual surface in said 
video image at said first orientation in relation to said actual surface" which would have 
been obvious to Heidmann's inserted objects and animation (column 3 line 56 to column 
4, line 14) because one of those inserted objects can be an annotation. 



Application/Control Number: 09/928,165 



Art Unit: 2671 



Page 15 



Claim 42 adds into claim 40 "storing a set of positions in an environment, said set 
of position bound a virtual advertisement, said step of converting said one or more 
positions includes transforming said set of positions in said environment to locations on 
said virtual surface; said step of converting said one or more locations includes 
determining where said set of positions are in said video image; and said step of 
enhancing includes adding said virtual advertisement into said video image where said 
set of positions are in said video image" which would have been obvious to Heidmann's 
inserted objects (column 4, lines 1-14) because the inserted object can be "a virtual 
advertisement". 

Claim 43 adds into claim 35 "camera sensors; a gatherer computer, said gatherer 
computer receives camera sensor data from said camera sensors; a time code inserter, 
said time code inserter receives video and adds time codes to said video; and a touch 
screen, said one or more processing devices include a first processor and a second 
processor, said first processor receives camera sensor data from said gatherer and 
video from said time code inserter, said first processor also receives annotation data 
from said touch screen, said annotation data relates to a set of positions on said touch 
screen, said first processor converts said positions on said touch screen to said one or 
more locations in relation to said virtual surface, said first processor sends drawing 
information to said second processor, said drawing information is based on said one or 
more positions in said video image, said second processor directs said video 
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modification hardware to add one or more annotation image to said video image, said 
annotation image corresponds to said annotation data, said step of converting said one < 
or more locations is based on said camera sensor data" which would have been 
obvious to Heidmann's inserted objects and animation (column 3 line 56 to column 4, 
line 14) because one of those inserted objects can be an annotation. 

Claim 44 adds into claim 35 "camera sensors, said camera sensors provide 
camera sensor data to said one or more processors, said step of converting said one or 
more locations is based on said camera sensor data" which would have been obvious to 
Heidmann's camera data (column 4, lines 4-8; column 12, lines 43-57) because the 
camera data can be collected through the sensors as claimed. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Phu K. Nguyen whose telephone number is (703) 305 - 
9796. The examiner can normally be reached on M-F 8:00-4:30. 

The fax phone number for the organization where this application or proceeding 
is assigned is (703) 872-9306. Any inquiry of a general nature or relating to the status 
of this application or proceeding should be directed to the receptionist whose telephone 
number is (703) 305-3800. 



Phu K. Nguyen 
August 25, 2003 
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